research-article

A Systematic Review on Software Robustness Assessment

Authors:
Nuno Laranjeiro

University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal

University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal

0000-0003-0011-9901
View Profile

,
João Agnelo

University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal

University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering, Coimbra, Portugal

0000-0002-6051-3516
View Profile

,
Jorge Bernardino

Polytechnic of Coimbra, Coimbra Institute of Engineering (ISEC), Portugal and University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Portugal

Polytechnic of Coimbra, Coimbra Institute of Engineering (ISEC), Portugal and University of Coimbra, Centre for Informatics and Systems of the University of Coimbra, Portugal

0000-0001-9660-2011
View Profile

Authors Info & Claims

ACM Computing Surveys Volume 54 Issue 4Article No.: 89pp 1–65https://doi.org/10.1145/3448977

Published:03 May 2021Publication History

ACM Computing Surveys

Editorial Notes

The authors have requested minor, non-substantive changes to the VoR and, in accordance with ACM policies, a Corrected VoR was published on May 6, 2021. For reference purposes the VoR may still be accessed via the Supplemental Material section on this page.

Abstract

Robustness is the degree to which a certain system or component can operate correctly in the presence of invalid inputs or stressful environmental conditions. With the increasing complexity and widespread use of computer systems, obtaining assurances regarding their robustness has become of vital importance. This survey discusses the state of the art on software robustness assessment, with emphasis on key aspects like types of systems being evaluated, assessment techniques used, the target of the techniques, the types of faults used, and how system behavior is classified. The survey concludes with the identification of gaps and open challenges related with robustness assessment.

Supplemental Material

Available for Download

pdf

3448977-vor.pdf (2.7 MB)

Version of Record for "A Systematic Review on Software Robustness Assessment" by Laranjeiro et al., ACM Computing Surveys, Vol.54, No.4.

References

Mithun Acharya, Tao Xie, and Jun Xu. 2006Mining interface specifications for generating checkable robustness properties. In Proceedings of the 17th International Symposium on Software Reliability Engineering (ISSRE’06). IEEE Computer Society, 311--320. DOI:https://doi.org/10.1109/ISSRE.2006.29Google ScholarDigital Library
Hussein Al-haj Ahmad, Yasser Sedaghat, and Mahin Moradiyan. 2019. LDSFI: A lightweight dynamic software-based fault injection. In Proceedings of the 9th International Conference on Computer and Knowledge Engineering (ICCKE’19). IEEE, 207--213. DOI:https://doi.org/10.1109/ICCKE48569.2019.8964875Google ScholarCross Ref
Arnaud Albinet, Jean Arlat, and Jean-Charles Fabre. 2004. Characterization of the impact of faulty drivers on the robustness of the Linux kernel. In Proceedings of the International Conference on Dependable Systems and Networks. IEEE, 867--876. DOI:https://doi.org/10.1109/DSN.2004.1311957Google ScholarCross Ref
Khaled Alnawasreh, Patrizio Pelliccione, Zhenxiao Hao, Mårten Rånge, and Antonia Bertolino. 2017. Online robustness testing of distributed embedded systems: An industrial approach. In Proceedings of the IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP’17). IEEE, 133--142. DOI:https://doi.org/10.1109/ICSE-SEIP.2017.17Google ScholarDigital Library
Fattoh Alqershi, Muhammad AL-Qurishi, Sk Md Mizanur Rahman, and Atif Alamri. 2014. Android vs. iOS: The security battle. In Proceedings of the World Congress on Computer Applications and Information Systems (WCCAIS’14). 1--8.Google ScholarCross Ref
Johan Arcile, Raymond Devillers, Hanna Klaudel, Witold Klaudel, and Bożena Woźna-Szcześniak. 2018. Modeling and checking robustness of communicating autonomous vehicles. In Proceedings of the 14th International Conference on Distributed Computing and Artificial Intelligence, Sigeru Omatu, Sara Rodríguez, Gabriel Villarrubia, Pedro Faria, Paweł Sitek, and Javier Prieto (Eds.). Springer International Publishing, Cham, 173--180. DOI:https://doi.org/10.1007/978-3-319-62410-5_21Google Scholar
Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Trans. Softw. Eng. Methodol. 28, 1 (Jan. 2019), 3:1--3:37. DOI:https://doi.org/10.1145/3293455Google ScholarDigital Library
Bruno Areias, Nuno Humberto, Lucas Guardalben, José Maria Fernandes, and Susana Sargento. 2018. Towards an automated flying drones platform. In Proceedings of the 4th International Conference on Vehicle Technology and Intelligent Transport Systems—Volume 1 (VEHITS’18). INSTICC, SciTePress, 529--536. DOI:https://doi.org/10.5220/0006792405290536Google ScholarCross Ref
Jean Arlat, Jean-Charles Fabre, Manuel Rodríguez, and Frederic Salles. 2002. Dependability of COTS microkernel-based systems. IEEE Trans. Comput. 51, 2 (2002), 138--163. DOI:https://doi.org/10.1109/12.980005Google ScholarDigital Library
Rob Ashmore, Radu Calinescu, and Colin Paterson. 2021. Assuring the machine learning lifecycle: Desiderata, methods, and challenges. ACM Computing Surveys. Retrieved from http://arxiv.org/abs/1905.04223.Google Scholar
Association for Computing Machinery. 2020. ACM Digital Library. Retrieved July 1, 2020 from https://dl.acm.org/.Google Scholar
Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API fuzzing. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE’20). 748--758. DOI:https://doi.org/10.1109/ICSE.2019.00083Google ScholarDigital Library
Algirdas Avizienis, Jean-Claude Laprie, Brian Randell, and Carl Landwehr. 2004. Basic concepts and taxonomy of dependable and secure computing. IEEE Trans. Depend. Sec. Comput. 1, 1 (2004), 11--33. DOI:https://ieeexplore.ieee.org/document/1335465.Google ScholarDigital Library
Denise Azevedo, Ana Ambrosio, and Marco Vieira. 2013. HLA middleware robustness and scalability evaluation in the context of satellite simulators. In Proceedings of the IEEE 19th Pacific Rim International Symposium on Dependable Computing. IEEE, 312--317. DOI:https://doi.org/10.1109/PRDC.2013.53Google ScholarDigital Library
Yamine Aït-Ameur, Gamze Bel, Frédéric Boniol, S. Pairault, and Virginie Wiels. 2003. Robustness analysis of avionics embedded systems. SIGPLAN Not. 38, 7 (Jun. 2003), 123--132. DOI:https://doi.org/10.1145/780731.780750Google ScholarDigital Library
Michael Barr and Anthony Massa. 2006. Programming Embedded Systems: With C and GNU Development Tools (2 ed.). O’Reilly Media, Inc.Google Scholar
Douglas K. Barry. 2013. Web Services, Service-Oriented Architectures, and Cloud Computing (2nd ed.). Morgan Kaufmann, San Francisco, CA.Google Scholar
Carlos Leandro Gomes Batista, Anderson Coelho Weller, Eliane Martins, and Fátima Mattiello-Francisco. 2019. Towards increasing nanosatellite subsystem robustness. Acta Astron. 156 (2019), 187--196. DOI:https://doi.org/10.1016/j.actaastro.2018.11.011Google ScholarCross Ref
Sebastian Bauersfeld and Tanja E. J. Vos. 2012. GUITest: A Java library for fully automated GUI robustness testing. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 330--333. DOI:https://doi.org/10.1145/2351676.2351739Google Scholar
Fevzi Belli, Axel Hollmann, and Weicheneric Wong. 2010. Towards scalable robustness testing. In Proceedings of the 4th International Conference on Secure Software Integration and Reliability Improvement. IEEE, 208--216. DOI:https://doi.org/10.1109/SSIRI.2010.14Google ScholarDigital Library
Mohamed N. Bennani and Daniel A. Menascé. 2004. Assessing the robustness of self-managing computer systems under highly variable workloads. In Proceedings of the International Conference on Autonomic Computing.IEEE Press, 62--69. DOI:https://doi.org/10.1109/ICAC.2004.1301348Google Scholar
Antonia Bertolino, Guglielmo De Angelis, Micael Gallego, Boni García, Francisco Gortázar, Francesca Lonetti, and Eda Marchetti. 2019. A systematic review on cloud testing. ACM Comput. Surv. 52, 5, Article 93 (Sep. 2019), 42 pages. DOI:https://doi.org/10.1145/3331447Google Scholar
Keshav Bimbraw. 2015. Autonomous cars: Past, present and future—A review of the developments in the last century, the present scenario and the expected future of autonomous vehicle technology. Proceedings of the 12th International Conference on Informatics in Control, Automation and Robotics (ICINCO’15), 191--198. DOI:https://doi.org/10.5220/0005540501910198Google ScholarDigital Library
Marcel Boehme, Cristian Cadar, and Abhik Roychoudhury. 2020. Fuzzing: Challenges and reflections. IEEE Software (2020). DOI:https://doi.org/10.1109/MS.2020.3016773Google Scholar
Pearl Brereton, Barbara A. Kitchenham, David Budgen, Mark Turner, and Mohamed Khalil. 2007. Lessons from applying the systematic literature review process within the software engineering domain. J. Syst. Softw. 80, 4 (2007), 571--583. DOI:https://doi.org/10.1016/j.jss.2006.07.009Google ScholarDigital Library
Jennifer Brings, Marian Daun, Kevin Keller, Patricia Aluko Obe, and Thorsten Weyer. 2020. A systematic map on verification and validation of emergent behavior in software engineering research. Fut. Gen. Comput. Syst. 112 (2020), 1010--1037. DOI:https://doi.org/10.1016/j.future.2020.06.049Google ScholarCross Ref
Pete Broadwell, Naveen Sastry, and Jonathan Traupman. 2002. FIG: A prototype tool for online verification of recovery. In Proceedings of the Workshop on Self-Healing, Adaptive and Self-Managed Systems.Google Scholar
Manfred Broy, Ingolf H. Kruger, Alexander Pretschner, and Christian Salzmann. 2007. Engineering automotive software. Proc. IEEE 95, 2 (2007), 356--373.Google ScholarCross Ref
Javier Cámara, Rogério de Lemos, Nuno Laranjeiro, Rafael Ventura, and Marco Vieira. 2014. Robustness evaluation of the rainbow framework for self-adaptation. In Proceedings of the 29th Annual ACM Symposium on Applied Computing (SAC’14). Association for Computing Machinery, New York, NY, 376--383. DOI:https://doi.org/10.1145/2554850.2554935Google ScholarDigital Library
Ilaria Canova Calori, Tor Stålhane, and Sven Ziemer. 2007. Robustness analysis using fmea and bbn: Case study for a web-based application. Proceedings of the 3rd International Conference on Web Information Systems and Technologies (Webist’07).Google Scholar
Wallace Cardoso, Eliane Martins, Nuno Laranjeiro, and Nuno Antunes. 2019. Combining state and interface - Based robustness testing for OpenStack components. In Proceedings of the 9th Latin-American Symposium on Dependable Computing (LADC’19). IEEE Press, 1--10. DOI:https://doi.org/10.1109/LADC48089.2019.8995720Google ScholarCross Ref
Wallace F. F. Cardoso and Eliane Martins. 2018. Using a search and model based framework to improve robustness tests in cloud platforms. In Proceedings of the III Brazilian Symposium on Systematic and Automated Software Testing (SAST’18). Association for Computing Machinery, New York, NY, 67--76. DOI:https://doi.org/10.1145/3266003.3266011Google Scholar
João Carreira, Henrique Madeira, and João G. Silva. 1998. Xception: A technique for the experimental evaluation of dependability in modern computers. IEEE Trans. Softw. Eng. 24, 2 (Feb. 1998), 125--136. DOI:https://doi.org/10.1109/32.666826Google ScholarDigital Library
Gabriella Carrozza, Aniello Napolitano, Nuno Laranjeiro, and Marco Vieira. 2011. WSRTesting: Hands-on solution to improve web services robustness testing. Proceedings of the Latin-American Symposium on Dependable Computing Workshops, 41--46. DOI:https://doi.org/10.1109/LADCW.2011.15Google ScholarCross Ref
Ana Cavalli, Eliane Martins, and Anderson Morais. 2008. Use of invariant properties to evaluate the results of fault-injection-based robustness testing of protocol implementations. In Proceedings of the IEEE International Conference on Software Testing Verification and Validation Workshop. IEEE Press, 21--30. DOI:https://doi.org/10.1109/ICSTW.2008.51Google ScholarDigital Library
Franck Chauvel, Hui Song, Nicolas Ferry, and Franck Fleurey. 2015. Evaluating robustness of cloud-based systems. J. Cloud Comput. 4, 1 (2015), 18.Google ScholarCross Ref
Zitao Chen, Niranjhana Narayanan, Bo Fang, Guanpeng Li, Karthik Pattabiraman, and Nathan DeBardeleben. 2020. TensorFI: A flexible fault injection framework for TensorFlow applications. In Proceedings of the 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE’20). 426--435. DOI:https://doi.org/10.1109/ISSRE5003.2020.00047ISSN: 2332-6549.Google ScholarCross Ref
F. Curbera, M. Duftler, R. Khalaf, W. Nagy, N. Mukhi, and S. Weerawarana. 2002. Unraveling the Web services web: An introduction to SOAP, WSDL, and UDDI. IEEE Internet Computing 6, 2 (2002), 86--93. DOI:10.1109/4236.991449Google ScholarDigital Library
Hoang-Nam Chu, Jean Arlat, Marc-Olivier Killijian, Benjamin Lussier, and David Powell. 2009. Robustness testing of robot controller software. In Proceedings of the 12th European Workshop on Dependable Computing (EWDC’09), Hélène Waeselynck (Ed.).Google Scholar
Sijun Chu and Hao Wu. 2018. Research on offense and defense technology for iOS kernel security mechanism. AIP Conf. Proc. 1955, 1 (2018), 040132. DOI:https://doi.org/10.1063/1.5033796Google ScholarCross Ref
Kai Cong, Li Lei, Zhenkun Yang, and Fei Xie. 2015. Automatic fault injection for driver robustness testing. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA’15). ACM, New York, NY, 361--372. DOI:https://doi.org/10.1145/2771783.2771811Google ScholarDigital Library
Diamantino Costa and Henrique Madeira. 1999. Experimental assessment of COTS DBMS robustness under transient faults. In Proceedings of the Pacific Rim International Symposium on Dependable Computing (PRDC’99). IEEE Computer Society, 1.Google ScholarCross Ref
Diamantino Costa, Tiago Rilho, and Henrique Madeira. 2000. Joint evaluation of performance and robustness of a COTS DBMS through fault-injection. In Proceedings of the 2000 International Conference on Dependable Systems and Networks (DSN’00). IEEE Computer Society, 251--260.Google ScholarCross Ref
Domenico Cotroneo, Domenico Di Leo, Roberto Natella, and Roberto Pietrantuono. 2011. A case study on state-based robustness testing of an operating system for the avionic domain. In Computer Safety, Reliability, and Security, Francesco Flammini, Sandro Bologna, and Valeria Vittorini (Eds.). Springer, Berlin, 213--227.Google Scholar
Domenico Cotroneo, Flavio Frattini, Roberto Pietrantuono, and Stefano Russo. 2015. State-based robustness testing of IaaS cloud platforms. In Proceedings of the 5th International Workshop on Cloud Data and Platforms (CloudDP’15).Google ScholarDigital Library
Domenico Cotroneo, Antonio Ken Iannillo, and Roberto Natella. 2019. Evolutionary fuzzing of Android OS vendor system services. Emp. Softw. Eng. 24, 6 (Dec. 2019), 3630--3658. DOI:https://doi.org/10.1007/s10664-019-09725-6Google Scholar
Domenico Cotroneo, Antonio Ken Iannillo, Roberto Natella, and Stefano Rosiello. 2019. Dependability assessment of the Android OS through fault injection. IEEE Trans. Reliabil. (2019), 1--16. DOI:https://doi.org/10.1109/TR.2019.2954384Google Scholar
Domenico Cotroneo, Domenico Di Leo, Francesco Fucci, and Roberto Natella. 2013. SABRINE: State-based robustness testing of operating systems. In Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering (ASE’13). IEEE, 125--135. DOI:https://doi.org/10.1109/ASE.2013.6693073Google ScholarDigital Library
Christoph Csallner and Yannis Smaragdakis. 2004. JCrasher: An automatic robustness tester for Java. Softw. Prac. Exp. 34 (09 2004), 1025--1050. DOI:https://doi.org/10.1002/spe.602Google Scholar
Javier Cámara, Rogério de Lemos, Nuno Laranjeiro, Rafael Ventura, and Marco Vieira. 2013. Robustness evaluation of controllers in self-adaptive software systems. In Proceedings of the 6th Latin-American Symposium on Dependable Computing. 1--10. DOI:https://doi.org/10.1109/LADC.2013.17Google ScholarDigital Library
Javier Cámara, Rogério de Lemos, Nuno Laranjeiro, Rafael Ventura, and Marco Vieira. 2014. Testing the robustness of controllers for self-adaptive systems. J. Brazil. Comput. Soc. 20 (2014). DOI:https://doi.org/10.1186/1678-4804-20-1Google Scholar
Javier Cámara, Rogério de Lemos, Nuno Laranjeiro, Rafael Ventura, and Marco Vieira. 2017. Robustness-driven resilience evaluation of self-adaptive software systems. IEEE Trans. Depend. Sec. Comput. 14, 1 (2017), 50--64.Google ScholarDigital Library
DBLP Team. 2020. DBLP. Retrieved July 1, 2020 from https://dblp.uni-trier.de/.Google Scholar
Harvey M. Deitel, Paul J. Deitel, and David R. Choffnes. 2003. Operating Systems (3rd Edition). Prentice Hall, Upper Saddle River, NJ.Google Scholar
Christopher P. Dingman, Joe Marshall, and Daniel P. Siewiorek. 1995. Measuring robustness of a fault tolerant aerospace system. In Proceedings of the 25th International Symposium on Fault-Tolerant Computing.522--527. DOI:https://doi.org/10.1109/FTCS.1995.466945Google Scholar
João Durães and Henrique Madeira. 2002. Characterization of operating systems behavior in the presence of faulty drivers through software fault emulation. In Proceedings of the Pacific Rim International Symposium on Dependable Computing. 201--209. DOI:https://doi.org/10.1109/PRDC.2002.1185639Google ScholarCross Ref
João Durães and Henrique Madeira. 2002. Emulation of software faults by educated mutations at machine-code level. In Proceedings of the 13th International Symposium on Software Reliability Engineering. IEEE Press, 329--340.Google ScholarCross Ref
João Durães and Henrique Madeira. 2006. Emulation of software faults: A field data study and a practical approach. IEEE Trans. Softw. Eng. 32, 11 (Nov. 2006), 849--867. DOI:https://doi.org/10.1109/TSE.2006.113Google ScholarDigital Library
Elsevier BV. 2020. Scopus. Retrieved July 1, 2020 from https://www.scopus.com/home.uri.Google Scholar
Huan Feng and Kang G. Shin. 2016. BinderCracker: Assessing the robustness of Android system services. arxiv:1604.06964.Retrieved from http://arxiv.org/abs/1604.06964.Google Scholar
Jean-Claude Fernandez, Laurent Mounier, and Cyril Pachon. 2005. A model-based approach for robustness testing. In Testing of Communicating Systems, Ferhat Khendek and Rachida Dssouli (Eds.). Springer, Berlin, 333--348.Google Scholar
Kimberly Fernsler and Philip John Koopman. 1999. Robustness testing of a distributed simulation backplane. In Proceedings of the 10th International Symposium on Software Reliability Engineering. 189--198. DOI:https://doi.org/10.1109/ISSRE.1999.809324Google ScholarDigital Library
Christof Fetzer and Zhen Xiao. 2005. Automatic testing for robustness violations. In Testing Commercial-off-the-Shelf Components and Systems. DOI:https://doi.org/10.1007/3-540-27071-X_16Google Scholar
Justin E. Forrester and Barton P. Miller. 2000. An empirical study of the robustness of Windows NT applications using random testing. In Proceedings of the 4th Conference on USENIX Windows Systems Symposium, Volume 4 (WSS’00). USENIX Association, Berkeley, CA, 6--6. http://dl.acm.org/citation.cfm?id=1267102.1267108Google Scholar
Hacène Fouchal, Antoine Rollet, and Abbas Tarhini. 2005. Robustness of composed timed systems. In Proceedings of the SOFSEM 2005: Theory and Practice of Computer Science, Peter Vojtáš, Mária Bieliková, Bernadette Charron-Bost, and Ondrej Sýkora (Eds.). Springer, Berlin, 157--166.Google ScholarDigital Library
Hacène Fouchal, Antoine Rollet, and Abbas Tarhini. 2010. Robustness testing of composed real-time systems. J, Comput, Methods Sci, Eng, 10, 1-2S2 (Sep. 2010), 135--148.Google Scholar
Chen Fu, Ana Milanova, Barbara Gershon Ryder, and David G. Wonnacott. 2005. Robustness testing of Java server applications. IEEE Trans. Softw. Eng. 31, 4 (Apr, 2005), 292--311. DOI:https://doi.org/10.1109/TSE.2005.51Google ScholarDigital Library
Yulong Fu and Ousmane Koné. 2014. Security and robustness by protocol testing. IEEE Syst, J, 8, 3 (Sep. 2014), 699--707. DOI:https://doi.org/10.1109/JSYST.2012.2223532Google ScholarCross Ref
Anup K. Ghosh and Matthew Schmid. 1999. An approach to testing COTS software for robustness to operating system exceptions and errors. In Proceedings of the 10th International Symposium on Software Reliability Engineering. 166--174. DOI:https://doi.org/10.1109/ISSRE.1999.809321Google Scholar
Anup K. Ghosh, Matt Schmid, and Viren Shah. 1998. Testing the robustness of Windows NT software. In Proceedings of the 9th International Symposium on Software Reliability Engineering. 231--235. DOI:https://doi.org/10.1109/ISSRE.1998.730886Google ScholarCross Ref
Cristiano Giuffrida, Anton Kuijsten, and Andrew S. Tanenbaum. 2013. EDFI: A dependable fault injection tool for dependability benchmarking experiments. In Proceedings of the IEEE 19th Pacific Rim International Symposium on Dependable Computing. 31--40. DOI:https://doi.org/10.1109/PRDC.2013.12Google ScholarDigital Library
Google. 2020. Google Scholar. Retrieved July 1, 2020 from https://scholar.google.com/.Google Scholar
Gustavo Grieco, Will Song, Artur Cygan, Josselin Feist, and Alex Groce. 2020. Echidna: Effective, usable, and fast fuzzing for smart contracts. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’20). Association for Computing Machinery, New York, NY, 557--560. DOI:https://doi.org/10.1145/3395363.3404366Google ScholarDigital Library
Samer Hanna and Malcolm Munro. 2008. An Approach for WSDL-Based Automated Robustness Testing of Web Services. 1093--1104. DOI:https://doi.org/10.1007/978-0-387-78578-3_40Google Scholar
Ahmet Hayran, Muratcan Igdeli, Atif Yilmaz, and Cemal Gemci. 2016. Security evaluation of IOS and Android. Int. J. Appl. Math. Electr. Comput. (2016), 258--261.Google ScholarCross Ref
Patrick Heckeler, Bastian Schlich, Thomas Kropf, G. Cardoso, Hanno Eichelberger, Jürgen Ruf, Stefan Huster, Sebastian Burg, and Wolfgang Rosenstiel. 2013. Accelerated robustness testing of state-based components using reverse execution. In Proceedings of the ACM Symposium on Applied Computing. DOI:https://doi.org/10.1145/2480362.2480587Google ScholarDigital Library
Casidhe Hutchison, Milda Zizyte, Patrick E. Lanigan, David Guttendorf, Michael Wagner, Claire Le Goues, and Philip Koopman. 2018. Robustness testing of autonomy software. In Proceedings of the IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP’18). 276--285.Google ScholarDigital Library
A. K. Iannillo, R. Natella, D. Cotroneo, and C. Nita-Rotaru. 2017. Chizpurfle: A gray-box android fuzzer for vendor service customizations. In Proceedings of the 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE’17). 1--11. DOI:https://doi.org/10.1109/ISSRE.2017.16Google ScholarCross Ref
IEEE. 1990. IEEE standard glossary of software engineering terminology. IEEE Std 610.12-1990 (Dec 1990), 1--84. DOI:https://doi.org/10.1109/IEEESTD.1990.101064Google Scholar
IEEE. 2020. IEEE Xplore. Retrieved July 1, 2020 from https://ieeexplore.ieee.org/Xplore/home.jsp.Google Scholar
Sylvia Ilieva, Denitsa Manova, Ilina Manova, Cesare Bartolini, Antonia Bertolino, and Francesca Lonetti. 2011. An automated approach to robustness testing of BPEL orchestrations. In Proceedings of the 2011 IEEE 6th International Symposium on Service Oriented System (SOSE’11). 193--203. DOI:https://doi.org/10.1109/SOSE.2011.6139108Google ScholarDigital Library
Tahar Jarboui, Jean Arlat, Yves Crouzet, Karama Kanoun, and Thomas Marteau. 2002. Analysis of the effects of real and injected software faults: Linux as a case study. In Proceedings of the 2002 Pacific Rim International Symposium on Dependable Computing.51--58. DOI:https://doi.org/10.1109/PRDC.2002.1185618Google ScholarCross Ref
Saurabh Jha, Subho S. Banerjee, James Cyriac, Zbigniew T. Kalbarczyk, and Ravishankar K. Iyer. 2018. AVFI: Fault injection for autonomous vehicles. In Proceedings of the 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W’18). 55--56. DOI:https://doi.org/10.1109/DSN-W.2018.00027Google ScholarCross Ref
Yue Jia and Mark Harman. 2011. An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37, 5 (Sep. 2011), 649--678. DOI:https://doi.org/10.1109/TSE.2010.62Google ScholarDigital Library
Zhen Ming Jiang and Ahmed E. Hassan. 2015. A survey on load testing of large-scale software systems. IEEE Trans. Softw. Eng. 41, 11 (Nov. 2015), 1091--1118. DOI:https://doi.org/10.1109/TSE.2015.2445340Google ScholarDigital Library
Ang Jin, Jianhui Jiang, Jiawei Hu, and Jungang Lou. 2008. A PIN-based dynamic software fault injection system. In Proceedings of the 9th International Conference for Young Computer Scientists. 2160--2167. DOI:https://doi.org/10.1109/ICYCS.2008.329Google ScholarDigital Library
Chuanming Jing, Zhiliang Wang, Xia Yin, and Jianping Wu. 2008. A formal approach to robustness testing of network protocol. In Network and Parallel Computing, Jian Cao, Minglu Li, Min-You Wu, and Jinjun Chen (Eds.). Springer, Berlin, 24--37.Google Scholar
Andréas Johansson and Neeraj Suri. 2005. Error propagation profiling of operating systems. In Proceedings of the 2005 International Conference on Dependable Systems and Networks (DSN’05). 86--95. DOI:https://doi.org/10.1109/DSN.2005.45ISSN: 2158-3927.Google ScholarDigital Library
Andréas Johansson, Neeraj Suri, and Brendan Murphy. 2007. On the impact of injection triggers for OS robustness evaluation. In Proceedings of the 18th IEEE International Symposium on Software Reliability (ISSRE’07). 127--126.Google ScholarDigital Library
Andréas Johansson, Neeraj Suri, and Brendan Murphy. 2007. On the selection of error model(s) for OS robustness evaluation. In Proceedings of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07). 502--511. DOI:https://doi.org/10.1109/DSN.2007.71ISSN: 2158-3927.Google ScholarDigital Library
William Johansson, Martin Svensson, Ulf E. Larson, Magnus Almgren, and Vincenzo Gulisano. 2014. T-Fuzz: Model-based fuzzing for robustness testing of telecommunication protocols. In Proceedings of the Verification and Validation 2014 IEEE 7th International Conference on Software Testing. 323--332. DOI:https://doi.org/10.1109/ICST.2014.45ISSN: 2159-4848.Google ScholarDigital Library
Capers Jones. 2013. The Technical and Social History of Software Engineering. Addison-Wesley. 2013036017Google ScholarDigital Library
Diane Jordan, John Evdemon, Alexandre Alves, Assaf Arkin, Sid Askary, Charlton Barreto, Ben Bloch, Francisco Curbera, Mark Ford, Yaron Goland, et al. 2007. Web services business process execution language version 2.0. OASIS Stand. 11, 120 (2007), 5.Google Scholar
Rauli Kaksonen, Marko Laakso, and Ari Takanen. 2001. Software Security Assessment through Specification Mutations and Fault Injection. Springer US, Boston, MA, 173--183. DOI:https://doi.org/10.1007/978-0-387-35413-2_16Google Scholar
Anatoli Kalysch, Mark Deutel, and Tilo Müller. 2020. Template-based Android inter process communication fuzzing. In Proceedings of the 15th International Conference on Availability, Reliability and Security (ARES’20). Association for Computing Machinery, New York, NY, 1--6. DOI:https://doi.org/10.1145/3407023.3407052Google ScholarDigital Library
Karama Kanoun, Yves Crouzet, Ali Kalakech, Ana-Elena Rugina, and Philippe Rumeau. 2005. Benchmarking the dependability of Windows and Linux using PostMark/spl trade/ workloads. In Proceedings of the 16th IEEE International Symposium on Software Reliability Engineering (ISSRE’05). 10--20. DOI:https://doi.org/10.1109/ISSRE.2005.13Google ScholarDigital Library
Deborah Katz, Milda Zizyte, Casidhe Hutchison, David Guttendorf, Patrick E. Lanigan, Eric Sample, Philip Koopman, Michael Wagner, and Claire Le Goues. 2020. Robustness inside out testing. In Proceedings of the 50th Annual IEEE-IFIP International Conference on Dependable Systems and Networks-Supplemental Volume. 1--4. DOI:https://doi.org/10.1109/DSN-S50200.2020.00013Google ScholarCross Ref
Deborah S. Katz, Casidhe Hutchison, Milda Zizyte, and Claire Le Goues. 2020. Detecting execution anomalies as an oracle for autonomy software robustness. In Proceedings of the International Conference on Robotics and Automation (ICRA’20). DOI:https://doi.org/10.1109/ICRA40945.2020.9197060.Google ScholarCross Ref
Charles Keating, Ralph Rogers, Resit Unal, David Dryer, Andres Sousa-Poza, Robert Safford, William Peterson, and Ghaith Rabadi. 2003. System of systems engineering. Eng. Manage. J. 15, 3 (2003), 36--45. DOI:https://doi.org/10.1080/10429247.2003.11415214Google ScholarCross Ref
Barbara Kitchenham. 2004. Procedures for performing systematic reviews. Keele Univ. 33 (08 2004).Google Scholar
Philip Koopman and John DeVale. 1999. Comparing the robustness of POSIX operating systems. In Proceedings of the 29th Annual International Symposium on Fault-Tolerant Computing (FTCS’99). IEEE Computer Society, Washington, DC, USA, 30--. http://dl.acm.org/citation.cfm?id=795672.796953Google ScholarCross Ref
Philip Koopman and John DeVale. 2000. The exception handling effectiveness of POSIX operating systems. IEEE Trans. Softw. Eng. 26, 9 (Sep. 2000), 837--848. DOI:https://doi.org/10.1109/32.877845Google ScholarDigital Library
Philip Koopman, John Sung, Christopher Dingman, Daniel Siewiorek, and Ted Marz. 1997. Comparing operating systems using robustness benchmarks. In Proceedings of the 16th Symposium on Reliable Distributed Systems (SRDS’97). IEEE Computer Society, USA, 72.Google ScholarCross Ref
Nathan P. Kropp, Philip J. Koopman, and Daniel P. Siewiorek. 1998. Automated robustness testing of off-the-shelf software components. In Proceedings of the 28th Annual International Symposium on Fault-Tolerant Computing (FTCS’98). IEEE Computer Society.Google Scholar
Seung Hak Kuk and Hyeon Soo Kim. 2009. Robustness testing framework for Web services composition. In Proceedings of the IEEE Asia-Pacific Services Computing Conference (APSCC’09). 319--324. DOI:https://doi.org/10.1109/APSCC.2009.5394106Google ScholarCross Ref
Andras Kövi and Zoltan Micskei. 2010. Robustness testing of standard specifications-based HA middleware. In Proceedings of the IEEE 30th International Conference on Distributed Computing Systems Workshops. 302--306. DOI:https://doi.org/10.1109/ICDCSW.2010.73Google ScholarDigital Library
Nuno Laranjeiro, João Agnelo, and Jorge Bernardino. 2021. A Systematic Review on Software Robustness Assessment—Dataset. DOI:https://doi.org/10.5281/zenodo.4436319Google Scholar
Nuno Laranjeiro, Salvador Canelas, and Marco Vieira. 2008. Wsrbench: An on-line tool for robustness benchmarking. In Proceedings of the 2008 IEEE International Conference on Services Computing, Volume 2 (SCC’08). IEEE Computer Society, 187--194. DOI:https://doi.org/10.1109/SCC.2008.123Google ScholarDigital Library
Nuno Laranjeiro, Camilo Gomez, Enrico Schiavone, Leonardo Montecchi, Manoel J. M. Carvalho, Paolo Lollini, and Zoltán Micskei. 2019. Addressing verification and validation challenges in future cyber-physical systems. In Proceedings of the 9th Latin-American Symposium on Dependable Computing (LADC’19). 1--2. DOI:https://doi.org/10.1109/LADC48089.2019.8995714Google ScholarCross Ref
Nuno Laranjeiro, Marco Vieira, and Henrique Madeira. 2008. Experimental robustness evaluation of JMS middleware. In Proceedings of the IEEE International Conference on Services Computing, Vol. 1. 119--126. DOI:https://doi.org/10.1109/SCC.2008.129Google ScholarDigital Library
Nuno Laranjeiro, Marco Vieira, and Henrique Madeira. 2012. A robustness testing approach for SOAP Web services. J. Internet Serv. Appl. 3, 2 (01 Sep. 2012), 215--232. DOI:https://doi.org/10.1007/s13174-012-0062-2Google ScholarCross Ref
Mathieu Lavallée, Pierre-N. Robillard, and Reza Mirsalari. 2014. Performing systematic literature reviews with novices: An iterative approach. IEEE Trans. Educ. 57, 3 (2014), 175--181.Google ScholarDigital Library
Edward A. Lee. 2008. Cyber physical systems: Design challenges. In Proceedings of the 11th IEEE International Symposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC’08). 363--369.Google ScholarDigital Library
Edward Ashford Lee and Sanjit Arunkumar Seshia. 2016. Introduction to Embedded Systems: A Cyber-Physical Systems Approach (2nd ed.). The MIT Press.Google Scholar
Bin Lei, Xuandong Li, Zhiming Liu, Charles Morisset, and Volker Stolz. 2010. Robustness testing for software components. Sci. Comput. Program. 75, 10 (2010), 879--897. DOI:https://doi.org/10.1016/j.scico.2010.02.005Google ScholarDigital Library
Bin Lei, Zhiming Liu, Charles Morisset, and Xuandong Li. 2010. State based robustness testing for components. Electron. Notes Theor. Comput. Sci. 260 (2010), 173--188.Google ScholarDigital Library
Guanpeng Li, Siva Kumar Sastry Hari, Michael Sullivan, Timothy Tsai, Karthik Pattabiraman, Joel Emer, and Stephen W. Keckler. 2017. Understanding error propagation in deep learning neural network (DNN) accelerators and applications. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC’17). ACM, New York, NY, 1--12. DOI:https://doi.org/10.1145/3126908.3126964Google Scholar
Baozheng Liu, Chao Zhang, Guang Gong, Yishun Zeng, Haifeng Ruan, and Jianwei Zhuge. 2020. FANS: Fuzzing Android native system services via automated interface analysis. In Proceedings of the 29th USENIX Security Symposium (USENIX Security 20). USENIX Association, 307--323.Google Scholar
Nicholas Looker, Malcolm Munro, and Jiudong xu. 2004. Simulating errors in web services. J. Simul. 5 (12 2004), 1473--8031.Google Scholar
Nik Looker, Malcolm Munro, and Jie Xu. 2004. WS-FIT: A tool for dependability analysis of Web services. In Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC’04), Vol. 2. 120--123 vol.2.Google ScholarCross Ref
Nik Looker, Malcolm Munro, and Jie Xu. 2005. A comparison of network level fault injection with code insertion. In Proceedings of the 29th Annual International Computer Software and Applications Conference (COMPSAC’05), Vol. 1. 479--484 Vol. 2. DOI:https://doi.org/10.1109/COMPSAC.2005.16Google ScholarDigital Library
Panagiotis Louridas. 2006. Static code analysis. IEEE Softw. 23, 4 (Jul. 2006), 58--61. DOI:https://doi.org/10.1109/MS.2006.114Google ScholarDigital Library
Yang Lu. 2019. The blockchain: State-of-the-art and research challenges. J. Industr. Inf. Integr. 15 (2019), 80--90. DOI:https://doi.org/10.1016/j.jii.2019.04.002Google ScholarCross Ref
Henrique Madeira, Raphael R. Some, Francisco Moreira, Diamantino Costa, and David A. Rennels. 2002. Experimental evaluation of a COTS system for space applications. In Proceedings of the International Conference on Dependable Systems and Networks. 325--330.Google Scholar
Ricardo Maia, Luis Henriques, Ricardo Barbosa, Diamantino Costa, and Henrique Madeira. 2005. Xception fault injection and robustness testing framework: A case-study of testing RTEMS. In VI Test and Fault Tolerance Workshop.Google Scholar
Amiya K. Maji, Fahad A. Arshad, Saurabh Bagchi, and Jan S. Rellermeyer. 2012. An empirical study of the robustness of Inter-component Communication in Android. In Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’12). 1--12. DOI:https://doi.org/10.1109/DSN.2012.6263963Google ScholarCross Ref
Valentin Jean Marie Manès, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2019. The art, science, and engineering of fuzzing: A survey. IEEE Trans. Softw. Eng. (2019). DOI:https://doi.org/10.1109/TSE.2019.2946563Google Scholar
Evan Martin, Suranjana Basu, and Tao Xie. 2007. Automated testing and response analysis of web services. In Proceedings of the IEEE International Conference on Web Services (ICWS’07). 647--654. DOI:https://doi.org/10.1109/ICWS.2007.49Google ScholarCross Ref
Evan Martin, Suranjana Basu, and Tao Xie. 2007. WebSob: A tool for robustness testing of web services. In Companion to the Proceedings of the 29th International Conference on Software Engineering (ICSE COMPANION’07). IEEE Computer Society, 65--66. DOI:https://doi.org/10.1109/ICSECOMPANION.2007.84Google Scholar
Eliane Martins, Cecília M. F. Rubira, and Nelson G. M. Leme. 2002. Jaca: A reflective fault injection tool based on patterns. In Proceedings of the International Conference on Dependable Systems and Networks. 483--487.Google Scholar
Fátima Mattiello-Francisco, Eliane Martins, Ana Rosa Cavalli, and Edgar Toshiro Yano. 2012. InRob: An approach for testing interoperability and robustness of real-time embedded software. J. Syst. Softw. 85 (01 2012), 3--15. DOI:https://doi.org/10.1016/j.jss.2011.02.034Google Scholar
Fatima Mattiello-Francisco, Eliane Martins, Andre Corsetti, Ana Rosa Cavalli, and Edgar Yano. 2009. Extended interoperability models for timed system robustness testing. In Proceedings of the IEEE Latin-American Conference on Communications. 1--6. DOI:https://doi.org/10.1109/LATINCOM.2009.5304903Google ScholarCross Ref
Naaliel Mendes, João Durães, and Henrique Madeira. 2010. Evaluating and comparing the impact of software faults on web servers. In Proceedings of the European Dependable Computing Conference. 33--42. DOI:https://doi.org/10.1109/EDCC.2010.14Google ScholarDigital Library
Manuel Mendonça and Nuno Ferreira Neves. 2007. Robustness testing of the Windows DDK. In Proceedings of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07), 554--564.Google ScholarDigital Library
Tim Menzies and Charles Pecheur. 2005. Verification and validation and artificial intelligence. In Advances in Computers, Vol. 65. Elsevier, 153--201. DOI:https://doi.org/10.1016/S0065-2458(05)65004-8Google Scholar
Zoltán Micskei, Henrique Madeira, Alberto Avritzer, István Majzik, Marco Vieira, and Nuno Antunes. 2012. Robustness testing techniques and tools. In Resilience Assessment and Evaluation of Computing Systems. Springer, 323--339.Google Scholar
Zoltán Micskei, István Majzik, and Francis Tam. 2006. Robustness testing techniques for high availability middleware solutions. In Proceedings of the International Workshop on Engineering of Fault Tolerant Systems (EFTS’06).Google Scholar
Zoltán Micskei, István Majzik, and Francis Tam. 2007. Comparing robustness of AIS-based middleware implementations. In Service Availability, Miroslaw Malek, Manfred Reitenspieß, and Aad van Moorsel (Eds.). Springer, Berlin, 20--30.Google Scholar
Barton P. Miller, Gregory Cooksey, and Fredrick Moore. 2006. An empirical study of the robustness of MacOS applications using random testing. In Proceedings of the 1st International Workshop on Random Testing (RT’06). Association for Computing Machinery, New York, NY, 46--54. DOI:https://doi.org/10.1145/1145735.1145743Google ScholarDigital Library
Barton P. Miller, Louis Fredriksen, and Bryan So. 1990. An empirical study of the reliability of UNIX utilities. Commun. ACM 33, 12 (Dec. 1990), 32--44. DOI:https://doi.org/10.1145/96267.96279Google ScholarDigital Library
Barton P. Miller, David Koski, Cjin Pheow, Lee Vivekananda Maganty, Ravi Murthy, Ajitkumar Natarajan, and Jeff Steidl. 1995. Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and Services. Technical Report.Google Scholar
Bartolomeo Montrucchio, Maurizio Rebaudengo, Alejandro, and David Velasco. 2014. Software-implemented fault injection in operating system kernel mutex data structure. In Proceedings of the IEEE 5th Latin American Symposium on Circuits and Systems. 1--6.Google ScholarCross Ref
Bartolomeo Montrucchio, Maurizio Rebaudengo, and Alejandrand David Velasco. 2014. Fault injection in the process descriptor of a Unix-based operating system. In Proceedings of the IEEE International Symposium on Defect and Fault Tolerance in VLSI and Nanotechnology Systems (DFT’14). 281--286.Google ScholarCross Ref
Regina Moraes, Ricardo Barbosa, João Durães, Naaliel Mendes, Eliane Martins, and Henrique Madeira. 2006. Injection of faults at component interfaces and inside the component code: Are they equivalent? In Proceedings of the Sixth European Dependable Computing Conference. 53--64.Google ScholarDigital Library
Arup Mukherjee and Daniel P. Siewiorek. 1997. Measuring software dependability by robustness benchmarking. IEEE Trans. Softw. Eng. 23, 6 (Jun. 1997), 366--378. DOI:https://doi.org/10.1109/32.601075Google ScholarDigital Library
Maha Naceur, Lilia Sfaxi, and Riadh Robbana. 2014. Robustness testing of secure wireless sensor networks. In Proceedings of the International Conference on Automation, Control, Engineering and Computer Science (ACECS’14).Google Scholar
Aniello Napolitano, Gabriella Carrozza, Antonio Bovenzi, and Christian Esposito. 2011. Automatic robustness assessment of DDS-compliant middleware. In Proceedings of the IEEE 17th Pacific Rim International Symposium on Dependable Computing. 302--307. DOI:https://doi.org/10.1109/PRDC.2011.51Google ScholarDigital Library
Roberto Natella, Domenico Cotroneo, and Henrique S. Madeira. 2016. Assessing dependability with software fault injection: A survey. Comput. Surv. 48, 3 (Feb. 2016), 1--55. DOI:https://doi.org/10.1145/2841425Google ScholarDigital Library
Andy Neumann, Nuno Laranjeiro, and Jorge Bernardino. 2018. An analysis of public REST web service APIs. IEEE Trans. Serv. Comput. (2018), 1--1. DOI:https://doi.org/10.1109/TSC.2018.2847344Google ScholarCross Ref
Tai D. Nguyen, Long H. Pham, Jun Sun, Yun Lin, and Quang Tran Minh. 2020. sFuzz: An efficient adaptive fuzzer for solidity smart contracts. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (ICSE’20). Association for Computing Machinery, New York, NY, 778--788. DOI:https://doi.org/10.1145/3377811.3380334Google ScholarDigital Library
Bogdan Nicolescu, N. Ignat, Yvon Savaria, and Gabriela Nicolescu. 2005. Sensitivity of real-time operating systems to transient faults: A case study for MicroC kernel. In Proceedings of the 8th European Conference on Radiation and Its Effects on Components and Systems. DOI:https://doi.org/10.1109/RADECS.2005.4365596Google ScholarCross Ref
Jie Xu Nik Looker. 2003. Assessing the dependability of SOAP RPC-based web services by fault injection. In Proceedings of the 9th IEEE International Workshop on Object-Oriented Real-Time Dependable Systems. 163--163. DOI:https://doi.org/10.1109/WORDS.2003.1267504Google ScholarCross Ref
Edward E. Ogheneovo. 2014. Software dysfunction: Why do software fail?J. Comput. Chem. 2014 (2014), 25--35.Google Scholar
János Oláh and István Majzik. 2009. A model based framework for specifying and executing fault injection experiments. In Proceedings of the 2009 4th International Conference on Dependability of Computer Systems (DEPCOS-RELCOMEX’09). IEEE Computer Society, Los Alamitos, CA, 107--114. DOI:https://doi.org/10.1109/DepCoS-RELCOMEX.2009.41Google ScholarDigital Library
Charles P. Shelton, Philip Koopman, and Kobey DeVale. 2000. Robustness testing of the Microsoft Win32 API. In Proceedings of the International conference on dependable systems and networks, 261--270. DOI:https://doi.org/10.1109/ICDSN.2000.857548Google ScholarCross Ref
Jiantao Pan, Philip Koopman, Yennun Huang, Robert Gruber, and Mimi Ling Jiang. 2001. Robustness testing and hardening of CORBA ORB implementations. In Proceedings of the International Conference on Dependable Systems and Networks. 141--150. DOI:https://doi.org/10.1109/DSN.2001.941400Google Scholar
Karthik Pattabiraman and Benjamin Zorn. 2010. DoDOM: Leveraging DOM invariants for web 2.0 application robustness testing. In Proceedings of the International Symposium on Software Reliability Engineering (ISSRE’10). DOI:https://doi.org/10.1109/ISSRE.2010.17Google ScholarDigital Library
Miroslav Popovic. 2006. Communication Protocol Engineering. CRC Press, Boca Raton, FL.Google Scholar
Miroslav Popovic and Jelena Kovacevic. 2007. A statistical approach to model-based robustness testing. In Proceedings of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS’07). 485--494. DOI:https://doi.org/10.1109/ECBS.2007.13Google ScholarDigital Library
David Powell, Jean Arlat, Hoang Nam Chu, Felix Ingrand, and Marc-Olivier Killijian. 2012. Testing the input timing robustness of real-time control software for autonomous systems. In Proceedings of the 9th European Dependable Computing Conference. 73--83. DOI:https://doi.org/10.1109/EDCC.2012.16Google ScholarDigital Library
Pan Qing-He, Hong Rong, and Pan Shu. 2010. A software-implemented fault injector on windows NT platform. Inf. Technol. J. 9 (03 2010). DOI:https://doi.org/10.3923/itj.2010.576.582Google Scholar
Issam Rabhi. 2012. Robustness testing of web services composition. In Proceedings of the IEEE 14th International Conference on High Performance Computing and Communication. 631--638. DOI:https://doi.org/10.1109/HPCC.2012.90Google ScholarDigital Library
Aurora Ramírez, José Raúl Romero, and Christopher L. Simons. 2019. A systematic review of interaction in search-based software engineering. IEEE Trans. Softw. Eng. 45, 8 (2019), 760--781.Google ScholarCross Ref
Brandon Reagen, Udit Gupta, Lillian Pentecost, Paul Whatmough, Sae Kyu Lee, Niamh Mulholland, David Brooks, and Gu-Yeon Wei. 2018. Ares: A framework for quantifying the resilience of deep neural networks. In Proceedings of the 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC’18). 1--6. DOI:https://doi.org/10.1109/DAC.2018.8465834Google ScholarCross Ref
Vincenzo Riccio, Gunel Jahangirova, Andrea Stocco, Nargiz Humbatova, Michael Weiss, and Paolo Tonella. 2020. Testing machine learning based systems: A systematic mapping. Empir. Softw. Eng. 25, 6 (Nov. 2020), 5193--5254. DOI:https://doi.org/10.1007/s10664-020-09881-0Google ScholarCross Ref
Manuel Rodríguez, Arnaud Albinet, and Jean Arlat. 2002. MAFALDA-RT: A tool for dependability assessment of real-time systems. In Proceedings of the International Conference on Dependable Systems and Networks. 267--272. DOI:https://doi.org/10.1109/DSN.2002.1028909Google ScholarCross Ref
Antoine Rollet and Fares Saad-Khorchef. 2007. A formal approach to test the robustness of embedded systems using behaviour analysis. In Proceedings of the 5th ACIS International Conference on Software Engineering Research, Management Applications (SERA’07). 667--674.Google ScholarDigital Library
Antoine Rollet and Salva. 2009. Testing robustness of communicating systems using ioco-based approach. In Proceedings of the IEEE Symposium on Computers and Communications. 67--72. DOI:https://doi.org/10.1109/ISCC.2009.5202405Google ScholarCross Ref
Antoine Rollet and Sebastien Salva. 2008. Two complementary approaches to test robustness of reactive systems. In Proceedings of the IEEE International Conference on Automation, Quality and Testing, Robotics, Vol. 1. 47--53. DOI:https://doi.org/10.1109/AQTR.2008.4588704Google ScholarDigital Library
Abu Hasnat Mohammad Rubaiyat, Yongming Qin, and Homa Alemzadeh. 2018. Experimental resilience assessment of an open-source driving agent. In Proceedings of the 2018 IEEE 23rd Pacific Rim International Symposium on Dependable Computing (PRDC’18). 54--63. DOI:https://doi.org/10.1109/PRDC.2018.00016ISSN: 2473-3105.Google ScholarCross Ref
Juan-Carlos Ruiz, Jose-Carlos Campelo, Pedro Gil, and Juan Pardo. 2005. On-chip debugging-based fault emulation for robustness evaluation of embedded software components. In Proceedings of the 11th Pacific Rim International Symposium on Dependable Computing (PRDC’05). IEEE Computer Society, Los Alamitos, CA, 57--64. DOI:https://doi.org/10.1109/PRDC.2005.44Google ScholarDigital Library
Marek Rychlý and Martin Žouželka. 2012. Fault injection for web-services. In Proceedings of the 14th International Conference on Enterprise Information Systems (ICEIS’12). INSTICC, SciTePress, 377--383. DOI:https://doi.org/10.5220/0004153003770383Google Scholar
Fares Saad-Khorchef, Antoine Rollet, and Richard Castanet. 2007. A framework and a tool for robustness testing of communicating software. In Proceedings of the 2007 ACM Symposium on Applied Computing (SAC’07). ACM, New York, NY, 1461--1466. DOI:https://doi.org/10.1145/1244002.1244315Google ScholarDigital Library
Marcelo Invert Palma Salas, Paulo Lício De Geus, and Eliane Martins. 2015. Security testing methodology for evaluation of web services robustness—Case: XML injection. In Proceedings of the IEEE World Congress on Services. 303--310. DOI:https://doi.org/10.1109/SERVICES.2015.53Google ScholarDigital Library
Mazeiar Salehie and Ladan Tahvildari. 2009. Self-adaptive software: Landscape and research challenges. ACM Trans. Auton. Adapt. Syst. 4, 2, Article 14 (May 2009), 42 pages. DOI:https://doi.org/10.1145/1516533.1516538Google ScholarDigital Library
Sébastien Salva and Issam Rabhi. 2009. Automatic web service robustness testing from WSDL descriptions. In Proceedings of the 12th European Workshop on Dependable Computing.Google Scholar
Sébastien Salva and Issam Rabhi. 2010. Stateful web service robustness. In Proceedings of the 5th International Conference on Internet and Web Applications and Services. 167--173. DOI:https://doi.org/10.1109/ICIW.2010.32Google ScholarDigital Library
Raimondas Sasnauskas and John Regehr. 2014. Intent fuzzer: Crafting intents of death. In Proceedings of the 2014 Joint International Workshop on Dynamic Analysis (WODA’14) and Software and System Performance Testing, Debugging, and Analytics (PERTEA’14). ACM, New York, NY, 1--5. DOI:https://doi.org/10.1145/2632168.2632169Google ScholarDigital Library
Matthew Schmid, Anup K. Ghosh, and Frank Hill. 2000. Techniques for evaluating the robustness of Windows NT software. In Proceedings of the DARPA Information Survivability Conference and Exposition. DISCEX, Vol. 2. 347--360. DOI:https://doi.org/10.1109/DISCEX.2000.821532Google Scholar
Syed Muhammad Ali Shah, Daniel Sundmark, Birgitta Lindström, and Sten F. Andler. 2016. Robustness testing of embedded software systems: An industrial interview study. IEEE Access 4 (2016), 1859--1871. DOI:https://doi.org/10.1109/ACCESS.2016.2544951Google ScholarCross Ref
Raheleh Shahpasand, Samad Paydar, Yasser Sedaghat, and Reza Ramezani. 2019. A state-aware approach for robustness testing of embedded real-time operating systems. J. Comput. Knowl. Eng. 2, 1 (2019).Google Scholar
Raheleh Shahpasand, Yasser Sedaghat, and Samad Paydar. 2016. Improving the stateful robustness testing of embedded real-time operating systems. In Proceedings of the 6th International Conference on Computer and Knowledge Engineering (ICCKE’16). 159--164. DOI:https://doi.org/10.1109/ICCKE.2016.7802133Google ScholarCross Ref
Ali Shahrokni and Robert Feldt. 2011. RobusTest: A framework for automated testing of software robustness. In Proceedings of the 18th Asia-Pacific Software Engineering Conference. 171--178. DOI:https://doi.org/10.1109/APSEC.2011.58Google ScholarDigital Library
Ali Shahrokni and Robert Feldt. 2013. A systematic review of software robustness. Inf. Softw. Technol. 55, 1 (2013), 1--17. DOI:https://doi.org/10.1016/j.infsof.2012.06.002Google ScholarDigital Library
Leon Shklar and Rich Rosen. 2009. Web Application Architecture: Principles, Protocols and Practices (2nd ed.). Wiley, Chichester, UK.Google Scholar
Reda Siblini and Nashat Mansour. 2005. Testing web services. In Proceedings of the 3rd ACS/IEEE International Conference on Computer Systems and Applications, 2005, 135. DOI:https://doi.org/10.1109/AICCSA.2005.1387124Google ScholarDigital Library
Daniel P. Siewiorek, John J. Hudak, Byung-Hoon Suh, and Zary Segall. 1993. Development of a benchmark to measure system robustness. In Proceedings of the 23rd International Symposium on Fault-Tolerant Computing (FTCS-23). 88--97.Google ScholarCross Ref
Theodore John Socolofsky and Claudia Jeanne Kale. 1991. TCP/IP Tutorial. RFC 1180. RFC Editor. Retrieved from http://www.rfc-editor.org/rfc/rfc1180.txt.Google Scholar
Springer Nature Switzerland AG. 2020. Springer Link. Retrieved July 1, 2020 from https://link.springer.com/.Google Scholar
Byung-Hoon Suh, John Hudak, Daniel Siewiorek, and Zary Segall. 1992. Development of a benchmark to measure system robustness: Experiences and lessons learned. In Proceedings of the 3rd International Symposium on Software Reliability Engineering. 237--245.Google Scholar
Abbas Tarhini, Antoine Rollet, and Hacène Fouchal. 2005. A pragmatic approach for testing robustness on real-time component based systems. In Proceedings of the 3rd ACS/IEEE International Conference on Computer Systems and Applications.143--150. DOI:https://doi.org/10.1109/AICCSA.2005.1387132Google ScholarCross Ref
Cumhur Erkan Tuncali, Georgios Fainekos, Hisahiro Ito, and James Kapinski. 2018. Simulation-based adversarial test generation for autonomous vehicles with machine learning components. In Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV’18). 1555--1562. DOI:https://doi.org/10.1109/IVS.2018.8500421Google ScholarCross Ref
Ekincan Ufuktepe and Tugkan Tuglular. 2018. Estimating software robustness in relation to input validation vulnerabilities using Bayesian networks. Softw. Qual. J. 26, 2 (2018), 455--489. DOI:https://doi.org/10.1007/s11219-017-9359-5Google ScholarDigital Library
Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A taxonomy of model-based testing approaches. Softw. Test. Verif. Reliabil. 22, 5 (2012), 297--312. DOI:https://doi.org/10.1002/stvr.456Google ScholarDigital Library
Arunchandar Vasan and A. M. Memon. 2004. ASPIRE: Automated systematic protocol implementation robustness evaluation. In Proceedings of the Australian Software Engineering Conference. 241--250. DOI:https://doi.org/10.1109/ASWEC.2004.1290477Google Scholar
Alejandro Velasco, Bartolomeo Montrucchio, and Maurizio Rebaudengo. 2016. KITO tool: A fault injection environment in Linux kernel data structures. Microelectr. Reliabil. 60 (2016). DOI:https://doi.org/10.1016/j.microrel.2016.02.011Google Scholar
Marco Vieira, Nuno Laranjeiro, and Henrique Madeira. 2007. Assessing robustness of web-services infrastructures. In Proceedings of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07). 131--136. DOI:https://doi.org/10.1109/DSN.2007.16Google ScholarDigital Library
Marco Vieira, Nuno Laranjeiro, and Henrique Madeira. 2007. Benchmarking the robustness of web services. In Proceedings of the 13th Pacific Rim International Symposium on Dependable Computing (PRDC’07). 322--329.Google ScholarDigital Library
Mark Vigder, W. Morven Gentleman, and John C. Dean. 1996. COTS Software Integration: State of the Art. Vol. 39190. National Research Council Canada, Institute for Information Technology.Google Scholar
Steve Vinoski. 2002. Where is middleware. IEEE Internet Comput. 6, 2 (2002), 83--85. DOI:https://doi.org/10.1109/4236.991448Google ScholarDigital Library
Aaron E. Walsh. 2002. UDDI, SOAP, and WSDL: The Web Services Specification Reference Book. Prentice Hall Professional Technical Reference.Google Scholar
N. Walters. 1995. Systems architecture and COTS integration. In Proceedings of the SEI/MCC Symposium on the Use of COTS in Systems Integration, SEI Special Report CMU/SEI-95-SR-007.Google Scholar
J. Wang, G. Dong, J. Sun, X. Wang, and P. Zhang. 2019. Adversarial sample detection for deep neural network through model mutation testing. In Proceedings of the 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE’19). 1245--1256. DOI:https://doi.org/10.1109/ICSE.2019.00126ISSN: 1558-1225.Google ScholarDigital Library
Stefan Winter, Thorsten Piper, Oliver Schwahn, Roberto Natella, Neeraj Suri, and Domenico Cotroneo. 2015. GRINDER: On reusability of fault injection tools. In Proceedings of the IEEE/ACM 10th International Workshop on Automation of Software Test. 75--79. DOI:https://doi.org/10.1109/AST.2015.22Google ScholarDigital Library
Stefan Winter, Constantin Sârbu, Neeraj Suri, and Brendan Murphy. 2011. The impact of fault models on software robustness evaluations. In Proceedings of the 33rd International Conference on Software Engineering (ICSE’11). ACM, 51--60. DOI:https://doi.org/10.1145/1985793.1985801Google ScholarDigital Library
Stefan Winter, Michael Tretter, Benjamin Sattler, and Neeraj Suri. 2013. simFI: From single to simultaneous software fault injections. In Proceedings of the 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’13). IEEE Computer Society, 1--12. DOI:https://doi.org/10.1109/DSN.2013.6575310Google ScholarDigital Library
Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering. ACM, 1--10. DOI:https://doi.org/10.1145/2601248.2601268Google ScholarDigital Library
Karl Wüst and Arthur Gervais. 2018. Do you need a blockchain? In Proceedings of the Crypto Valley Conference on Blockchain Technology (CVCBT’18). 45--54.Google ScholarCross Ref
Valentin Wüstholz and Maria Christakis. 2020. Harvey: A greybox fuzzer for smart contracts. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’20). ACM, New York, NY, 1398--1409. DOI:https://doi.org/10.1145/3368089.3417064Google ScholarDigital Library
Valentin Wüstholz and Maria Christakis. 2020. Targeted greybox fuzzing with static lookahead analysis. In Proceedings of the 42nd International Conference on Software Engineering (ICSE’20). ACM, 789--800. DOI:https://doi.org/10.1145/3377811.3380388Google ScholarDigital Library
Lin Xiang, Zhan Zhang, Decheng Zuo, and Xiaozong Yang. 2013. Multi-layered system robustness testing strategy based on abnormal parameter. J. Comput. 8 (2013). DOI:https://doi.org/10.4304/jcp.8.7.1882-1891Google Scholar
Yang Xiang, Zhiliang Wang, and Xia Yin. 2009. SIP robustness testing based on TTCN-3. In Proceedings of the International Conference on Advanced Information Networking and Applications Workshops. 122--128. DOI:https://doi.org/10.1109/WAINA.2009.19Google ScholarDigital Library
Xiaofei Xie, Lei Ma, Felix Juefei-Xu, Minhui Xue, Hongxu Chen, Yang Liu, Jianjun Zhao, Bo Li, Jianxiong Yin, and Simon See. 2019. DeepHunter: A coverage-guided fuzz testing framework for deep neural networks. In Proceedings of the 28th ACM SIGSOFT Int. Symposium on Software Testing and Analysis (ISSTA’19). ACM, 146--157. DOI:https://doi.org/10.1145/3293882.3330579Google ScholarDigital Library
Luo Xu, Ji Wu, and Chao Liu. 2009. TTCN-3 based robustness test generation and automation. In Proceedings of the International Conference on Information Technology and Computer Science, Vol. 2. 120--125. DOI:https://doi.org/10.1109/ITCS.2009.164Google ScholarDigital Library
Shunkun Yang, Bin Liu, Shihai Wang, and Minyan Lu. 2013. Model-based robustness testing for avionics-embedded software. Chin. J. Aeronaut. 26, 3 (2013), 730--740. DOI:https://doi.org/10.1016/j.cja.2013.05.004Google ScholarCross Ref
Hui Ye, Shaoyin Cheng, Lanbo Zhang, and Fan Jiang. 2013. DroidFuzzer: Fuzzing the Android apps with intent-filter tag. In Proceedings of the International Conference on Advances in Mobile Computing & Multimedia (MoMM’13). Association for Computing Machinery, New York, NY, 68--74. DOI:https://doi.org/10.1145/2536853.2536881Google ScholarDigital Library
Edgardo Barsallo Yi, Amiya Maji, and Saurabh Bagchi. 2018. How reliable is my wearable: A fuzz testing-based study. In Proceedings of the 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’18). 410--417. DOI:https://doi.org/10.1109/DSN.2018.00050Google Scholar
Edgardo Barsallo Yi, Heng Zhang, Amiya K. Maji, Kefan Xu, and Saurabh Bagchi. 2020. Vulcan: Lessons on reliability of wearables through state-aware fuzzing. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services (MobiSys’20). ACM, New York, NY, 391--403. DOI:https://doi.org/10.1145/3386901.3388916Google ScholarDigital Library
Kamal Z. Zamli, Mohd Daud Alang Hassan, Nor Ashidi Mat Isa, and Siti Norbaya Azizan. 2006. An automated software fault injection tool for robustness assessment of java COTs. In Proceedings of the International Conference on Computing Informatics. 1--6. DOI:https://doi.org/10.1109/ICOCI.2006.5276549Google ScholarCross Ref
Jie M. Zhang, Mark Harman, Lei Ma, and Yang Liu. 2020. Machine learning testing: Survey, landscapes and horizons. IEEE Trans. Softw. Eng. (2020). DOI:https://doi.org/10.1109/TSE.2019.2962027Google ScholarDigital Library
Zhengmao Zhou, Yun Zhou, Ming Cai, and Lei Sun. 2013. A workload model based approach to evaluate the robustness of real-time operating system. In Proceedings of the IEEE 10th International Conference on High Performance Computing and Communications. 2027--2033. DOI:https://doi.org/10.1109/HPCC.and.EUC.2013.292Google ScholarCross Ref

Index Terms

A Systematic Review on Software Robustness Assessment
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
  2. Software organization and properties
    1. Extra-functional properties
      1. Software reliability

Recommendations

A systematic review of software robustness

Context: With the increased use of software for running key functions in modern society it is of utmost importance to understand software robustness and how to support it. Although there have been many contributions to the field there is a lack of a ...
Read More
A behavioral notion of robustness for software systems
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Software systems are designed and implemented with assumptions about the environment. However, once the system is deployed, the actual environment may deviate from its expected behavior, possibly undermining desired properties of the system. To enable ...
Read More
Adaptive robustness evaluation for complex system prognostics and health management software platform
Abstract
With the quantity and complexity of various complex systems have increased, their safety and reliability face many challenges. The prognostics and health management (PHM) software platform for complex systems is one of the important ...
Highlights
- The robustness indicators are obtained for complex system PHM software platform.
Read More

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Published in
ACM Computing Surveys Volume 54, Issue 4
May 2022
782 pages
ISSN:0360-0300
EISSN:1557-7341
DOI:10.1145/3464463
Editor:
Albert Zomaya
University of Sydney, Australia
Issue’s Table of Contents
Copyright © 2021 ACM
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
In-Cooperation
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
- Published: 3 May 2021
- Accepted: 1 February 2021
- Revised: 1 January 2021
- Received: 1 July 2020
Published in csur Volume 54, Issue 4

Permissions
Request permissions about this article.
Request Permissions

Check for updates
Author Tags
Software robustness
robustness evaluation
robustness testing
Qualifiers
- research-article
- Research
- Refereed
Conference
Funding Sources
Other Metrics
View Article Metrics

Article Metrics
- 5
  Total Citations
  View Citations
- 700
  Total Downloads
- Downloads (Last 12 months)160
- Downloads (Last 6 weeks)22
Other Metrics
View Author Metrics
Cited By
View all

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format .

View HTML Format

A Systematic Review on Software Robustness Assessment

ACM Computing Surveys

Editorial Notes

Abstract

Supplemental Material

Available for Download

References

Cited By

Index Terms

Recommendations

A systematic review of software robustness

A behavioral notion of robustness for software systems

Adaptive robustness evaluation for complex system prognostics and health management software platform